Learning Optimal Parameter Values in Dynamic Environment: An Experiment with Softmax Reinforcement Learning Algorithm

1. Introduction Many learning and heuristic search algorithms require tuning of parameters to achieve optimum performance. In stationary and deterministic problem domains this is usually achieved through off-line sensitivity analysis. However, this method breaks down in non-stationary and non-deterministic environments, where the optimal set of values for the parameters keep changing over time. What is needed in such scenarios is a meta-learning (ML) mechanism that can learn the optimal set of parameters on-line while the learning algor ithm is trying to learn its target concept. In this paper we present a simple meta-learning algorithm to learn the temperature parameter of the Softmax reinforcement-learning (RL) algorithm. We test the effectiveness of this meta-learning algorithm in two domains. The first is the classic reinforcement learning problem known as the k-armed bandit problem. The second domain is a stylized problem in e-procurement. It involves a context of strategic interaction consisting of homogeneous sellers of a single raw material or component vying for business from a single buyer. The sellers are modeled as artificial agents that learn increasingly effective bidding strategies. We model non-stationarity in the first domain of k-armed bandit problem by periodically switching the reward distributions. The second domain is in effect a non-stationary and non-deterministic learning problem since it is a game of strategic interaction involving multiple agents. We model one of the sellers as using the Softmax RL algorithm. We consider the other seller to be using one of several different learning algorithms. We show that the best value of the temperature parameter for the Softmax agent varies depending on the learning algorithm used by the other seller. In both domains we show the improvement in performance brought on by the use of meta-learning.